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ABSTRACT 



Noting that assessment continues to be at the forefront of 
the speech communication discipline and the academy in general, this paper 
reflects on one speech communication department's struggle to make assessment 
meaningful and doable. The Speech Communications Department at the University 
of Arkansas at Little Rock uses such "low tech" assessment methods as 
portfolios, yet the department finds itself in the throes of learning "high 
tech" options. The paper provides a case study on "high tech" assessment by 
first describing the context and background of the department’s assessment 
efforts. The remaining sections of the paper are devoted to describing two 
high tech resources the department is using and the specific lessons learned 
about these assessment methods. An appendix presents a timetable for 
assessing basic course learning objectives. (RS) 



**************++++++++*★★★★★★★★★★★★★★★★★★★★★★★★★★★★****★**+*****★***********+ 
* Reproductions supplied by EDRS are the best that can be made 

from the original document. 



* 



ED 427 381 



State of the Art Learning Labs and Assessment 



Gerald W. Driskill 
Speech Communication 
University of Arkansas, Little Rock 
2801 S. University Dr. 

Little Rock , AR 72204 
501-569-3158 
E-mail: gwdriskill@ualr.edu 



Paper presented to the National Communication Association, November 20-22, 
1998, New York, New York. 



o» 

0 

0 




U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

BLThis document has been reproduced as 
f received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



BEST COPY AVAILABLE 

2 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

& 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



y 



1 



Learning Labs and Assessment p. 1 



State of the Art Learning Labs and Assessment 

Assessment continues to be at the forefront of our discipline and the 
academy in general (Boyer, 1990; Rogers & Mentkowski, 1996). The past 
decade has seen continued pressure from employers, politicians, and citizens 
who have doubts about the effectiveness of the academy in preparing 
individuals for the complex world of work (Ewell, 1998). Faculty concerns with 
assessment frequently center on making assessment a tool to improve student 
learning while simultaneously finding ways to integrate various assessment 
methods and feedback into an already burdened schedule (Hugenberg, 1995; 
Mader, 1995). 

This paper reflects one department’s struggle to make assessment 
meaningful and doable. While our department uses such “low tech” 
assessment methods as portfolios, we find ourselves in the throes of learning 
“high tech” options. This paper provides a case study on “high tech” 
assessment by first describing the context and background of our assessment 
efforts. The remaining sections of the paper are devoted to describing two high 
tech resources we are using and the specific lessons we are learning about 
these assessment methods. 

Background and Context 

The Speech Communication Department at the University of Arkansas at 
Little Rock has been on the road toward formal assessment processes for the 
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past ten years. Our university engaged in process of first identifying core 
competencies expected of students who completed core courses (e.g, oral and 
written communication, critical thinking, etc.). Departments with core courses 
were then expected to develop assessment plans that translated these core 
competencies into general and specific learning objectives and appropriate 
plans for assessment of these objectives. By the year 1999-2000, all programs 
(not just those with core courses), are expected to not only have plans in place, 
but results of feedback from their assessment efforts available to the campus 
community. 

Before the university wide initiatives on assessment had begun, our 
department was moving in the direction of improving our program through 
assessing student performance. For example, we developed a capstone 
course that included a standardized evaluation of a senior level 
paper/presentation. Feedback from these performances has resulted in major 
changes to our curriculum and greater communication concerning learning 
objectives in all of our courses. 

Our assessment efforts are integrated with our already learning lab that 
relies on interns and computer based instruction (CBI) to assist students across 
campus and in our core or basic course. Our goal in developing the lab was to 
provide more time in the class for active learning strategies. The lab continues 
to be central to our efforts in that with a staff of graduate and undergraduate 
interns it is a source of both high touch and high tech efforts to promote oral 
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communication skills for students across all majors. It is in the context of this lab 
that our primary assessment activities take place. 

The Lab 

The Jerry P. Butler Speech Communication Interactive Learning Lab 
(better know as the SCIL Center) houses 25 Pentium PC's running off of an 
Windows NT server, all utilizing “Authorware Professional” to create lessons. 
Our use of these computers for assessment has involved three efforts. The first 
two efforts focus on assessment of our core course while the last effort is in 
place to enhance student performance on traditional tests as well as the post 
assessment. 

Assessment of Knowledge . We have used the computers to collect data 
on a standard pre and post 50-item multiple choice assessment instrument. 

This instrument is currently being revised through the use of software that will 
allow us to: (a) provide the assessment on the internet so that we may assess 
students at other points in time; (b) focus on one or more learning objectives by 
selecting out key items for analysis; (c) create a larger bank of items that will 
then insure students receive equivalent but different assessment questions; 
and (d) allow us to interface with SPSS software for pre and post assessment 
analysis. 1 

Assessment of Apprehension We also use the PRCA to gain pre and 
post data on communication apprehension. These data are not only used to 

1 The software is produced by Logic extension Resources, 7168 Archibald Ave, Ste. 240, Alta Loma, CA 
91701-6421. Voice: 909-980-0046; Fax: (909) 987-8706; Web: http://www.lxrtest.com. The name of 
the software is “LXR-TEXT 5.1.9. The price as of 8/19/98 was $1597.00. If you questions about the 
product, you may contact A1 De Marchi: E-mail: ademarchi@lxrtest.com; (909) 980-0046, xllO. 



Learning Labs and Assessment p. 4 



determine changes in apprehension levels as a result of our course, but also 
scores are provided to individuals students and their faculty in order to 
determine specific challenges the student may need assistance with during the 
semester. 

Assessment bv Chapter . We are developing chapter reviews using 
“Authorware” software that will give the students immediate feedback 
concerning the knowledge of chapter concepts. These assessments were 
developed as a way to enhance student performance in that students may use 
these modules before a class room test as well as for review of concepts missed 
after a test is completed. In some cases, faculty require students to complete 
chapter reviews if they performed poorly on an exam. 

The Decision Support Center 

Our second high tech resource arrived in our department about three 
years ago when our department was charged with maintaining and providing 
moderators/facilitators for an electronic meeting room. The Decision Support 
Center is set up with recessed computer monitors, each having a keyboard 
linked to a central screen that allows all parties to express their ideas 
simultaneously and anonymously. A few benefits of this technology include the 
way it: 

*Cuts meeting time by 60% 

‘Displays statistical analysis for prioritizing and generating consensus 
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‘Provides written and electronic records of all processes 2 
While this room is host to on and off campus groups for purposes ranging from 
brainstorming to strategic planning, the technology has also proven useful for 
assessment in two ways. 

Developing a Reliable Evaluation Instrument . For the last several years 
our department has been devising a speech evaluation instrument that all 
faculty would agree to use in their classes. Determining the form and content of 
this instrument has been a slowly evolving process with the first stages 
involving consensus on criteria (e.g, use of sources, clear points, etc.) and the 
later stages involving agreement on categories for rating each criteria. 

However, the final stage, that of determining the reliability of the instrument, has 
been made easier by the use of the electronic meeting room. The process 
involves (a) viewing a taped speech, (b) rating the speech on the each of the 
seven criteria, and providing a grade, and (c) discussion of the results. 

For example, in our first effort to use the software for reliability 
development, within seconds after scoring, we had a screen to view the degree 
of consensus on each of the items (See Table 1). 



2 The software, “GroupSystems,” is produced by Ventana Corporation. 1430 E. Ft. Lowell, Tuscon, AZ 
85719. 520-325-8228 (voice); 520-325-8319. 
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Table 1 : 



Table of Ratings from Decision Support Center 
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The above screen prompted a discussion of the reasons for the low 
consensus on three of the criteria (items #1 , 3, 4). We were able to clarify rating 
categories/terms and determine needed changes in order to move toward 
greater consistency in the way we use the form. We plan to hold at least one 
more reliability check round and then move toward using the center as a place 
to train new instructors in using the instrument. 

Focus Groups The meeting room has also been useful in collecting 
information from alumni of our program. The software can be used to create a 
survey or to simply gain information about program strengths, weaknesses, 
and recommendations. The value of the software is such data can then be 
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electronically collapsed into categories and made readily accessible as a 
document that would serve as a launching pad for a strategic planning session. 
In short, the Decision Center provides a way to make the move from data 
collection to planning seamless. 

Lessons Learned 

The following four lessons highlight just a few of the conclusions we 
have drawn from our high tech approach to assessment. We clearly have more 
to learn and have come to accept that we will be making adjustments in our 
process on a continuous basis. 

1. The high tech component of our assessment efforts provides a 
way to enhance oral communication across the curriculum efforts. 

Our department is committed to supporting faculty across campus in developing 
the communication skills of their students. The assessment software in the 
SOIL Center provides local and internet access that will enable us to assess 
student knowledge of communication across campus after they have left the 
basic course. Our long range plans include collecting data from a sample of 
such students to determine their retention of information related to our core 
learning objectives. In time, we hope to provide a link to needed tutorials/CBI 
materials that would be linked to a student’s education needs. These efforts 
provide the greatest promise for creating a university that engages students and 
faculty in discovering greater coherence in the curriculum (c.f., Tint, 1993) . 
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2. Success with high tech efforts in electronic meeting rooms 
assume the use of a skilled moderator and a faculty willing to 
collaborate and compromise. 

As indicated, the electronic meeting room provides immediate feedback. 
At first glance, such a vehicle seems like a dream come true (i.e., instantaneous 
graphs, tables, etc. of data). However, we have learned that the facilitator must 
be sure to follow several guides. First, limit the time for each meeting. Not all of 
the data should be sorted and discussed at one time. We rarely allow anyone 
to use the room for more than 3 hours in a day. The fact that 5 to 6 hours of 
work may be done in that time means that participants are generally overly 
fatigued if they are kept longer. 

Second, make the goals of the session clear. While the electronic 
meeting room allows for more efficient meetings, it can be less productive if a 
group is not clear on what can be done in the allotted time. Massive data at your 
finger tips does not mean a group can arrive in that same amount of time on 
what to do with the data. For example, we only discussed the three items on 
which their was a lack of consensus. 

Finally, if you are not part of a faculty that enjoys the process of sorting 
data, collaborating, and making creative use of conflict, then such a center may 
only create more problems. The fact that “votes” can be caste anonymously 
does allow for fuller and richer participant. However, in the case of the 
essential discussion of ideas and views related to the data, a department must 
have a everyone on the same page in terms of a supportive climate. We have 
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found that the anonymity soon breaks down as viewpoints are expressed and 
reasons are given for various ratings. 

3. High tech efforts must be guided by a “doable” assessment plan 
(or how “High Tech” has taught us to redefine data overload). Data 
from the earlier pilot version of our 50-item assessment instrument as well as 
the PRCA quickly stacked up as one beleaguered faculty member hustled to 
first relearn SPSS on the main frame and then SPSS for the PC. The data, 
once crunched, was unmanageable in that it was not guided by a coherent and 
systematically prioritized set of questions. We had data that would answer a 
dozen potentially useful questions. However, even though I teach 
organizational assessment, I fell prey to the ’’just get the data” pitfall. Because 
we were not clear as to which questions needed to be answered, we made little 
productive useful of the data. For example, I generated a table that showed 
score differences across all sections of the basic course as well as section by 
section. Impressive as this data was, our department never had the discussion 
as to how we might use such information. Our current plan calls for us to focus 
on one learning objective each year. Since we do not have a department or 
one person with the time to collect and evaluate data, we needed a plan that 
would allow us to limit our data collection to one or two focused questions each 
year. Appendix A provides a graph of our plan to rotate learning objectives 
over the coming years. 

4. Without low tech data we would have very little to show from all 
of our high tech efforts. The same year we gained data about section 
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differences from our SCIL Center, we also used a portfolio method to examine 
our learning objectives. This data allowed us to notice differences in what 
students (and faculty) perceived to be our expectations for our small group 
component of the course. Such information was crucial in guiding changes we 
made in standardizing assignments and communicating expectations across 
the department. The statistical data alone would not have provided such a clear 
direction. At best, it would have helped us to notice differences in scores on 
the component of the course, but explanation for those differences required a 
different method. 

Summary 

High tech assessment, like all technological solutions, holds great power 
and promise. Our efforts at the University of Arkansas at Little Rock have 
reflected the potential of these efforts to create a feedback loop for program 
improvement as well as a learning community with enhanced oral 
communication across the curriculum efforts. While a number of other 
possibilities exist due to the capacity to link our Lab and Decision Center to the 
Net, we currently find ourselves committed to improving the quality of what we 
do by limiting the focus of our data collection efforts, recognizing the 
importance of our low tech efforts, and improving what we do with assessment 
with each step we take. 
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APPENDIX A 

TIME TABLE FOR ASSESSING BASIC COURSE LEARNING OBJECTIVES 
SPEECH COMMUNICATION DEPARTMENT, 

UNIVERSITY OF ARKANSAS, LITTLE ROCK 



TIME TABLE FOR ASSESSING SPEECH 


COMMUNICATION 1300 


LEARNING OBJECTIVE 


YEAR 


# identify principles of effective participation in 
a small group problem-solving discussion. 


1997-98 


# identify the criteria for an effective speech and 
be able to effectively deliver a simulated in-class 


1998-98 


report. 

# identify the basic fallacies that appear in 
argumentative discourse. 


1999-00 


# identify the methods for effectively managing 
a conflict situation. 


2000-01 


# identify concepts and terms 

associated with oral communication (e.g., basic 

components of the communication process, etc.). 


2001-02 
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